DOCUMENT RESUME 



SP 039 783 

Parrish, Paige; Keyes, Marian; Cowley, Kimberly S. 

Study of Writing Instruction in Kentucky Schools: Evaluation 
of the Facilitator Training in the School Study of Writing 
Instruction . 

Appalachia Educational Lab., Charleston, WV. ; Kentucky State 
Dept, of Education, Frankfort. 

Office of Educational Research and Improvement (ED) , 
Washington, DC. 

2000 - 00-00 
43p . 

RJ96006001 

AEL, Inc., P.O. Box 1348, Charleston, WV 25325-1348; Tel: 
304-347-0400 or 800-624-9120; Fax: 304-347-0487; Email: 
aelinfo@ael.org; Web site: http://www.ael.org. 

Reports - Research (143) 

MF01/PC02 Plus Postage. 

Educational Improvement; Elementary Secondary Education; 
*Needs Assessment; *Portf olios (Background Materials); 
Program Effectiveness; Program Evaluation; *Writing 
Instruction; *Writing Skills 
Kentucky Education Reform Act 1990 

One mandate of the Kentucky Education Reform Act of 1990 is 
that students develop a writing portfolio. The Kentucky Department of 
Education and AEL, Inc., collaborated to help schools discern areas for 
improvement in their writing programs, then developed a needs assessment 
process instrument, the School Study of Writing Instruction (SSWI) . This 
report summarizes evaluative data collected from participants in 1999 at 
sessions to train them as potential facilitators for schools interested in 
conducting a SSWI, a process in the form of a handbook which lets schools 
compare different perspectives about the writing portfolio program. Data from 
interviews, reports, and evaluation forms indicate that participants 
considered the SSWI a well-designed, research-based, self-explanatory tool 
that schools could use in improvement efforts targeting writing instruction. 
They seemed pleased with the potential benefits of the process and felt the 
handbook had great utility for consolidation planning and needs assessment. 
The training sessions were well -received and successfully accomplished their 
purpose of introducing the research findings and study process, preparing 
participants to promote use of the SSWI in schools, and instructing 
participants in required tasks. Appended are a training agenda, evaluation 
forms, and a completed evaluation standards checklist. (SM) 



ED 450 108 

AUTHOR 

TITLE 



INSTITUTION 

SPONS AGENCY 

PUB DATE 
NOTE 
CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 

ABSTRACT 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



00 

o 



o 



Q 



W 



Evaluation of the Facilitator Training in the 
School Study of Writing Instruction 



/ 




A Collaboration between AEL, Inc. 
and 

Kentucky Department of Education 



$ 

r+- 

cr 

o 

f Q 

\mc 




BEST COPY AVA5LABLE 



2 



■p 



U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATIC 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Evaluation of the Facilitator Training in the 
School Study of Writing Instruction 



by 

Paige Parrish 
Marian Keyes 
Kimberly S. Cowley 



July 2000 



AEL, Inc. 

Post Office Box 1348 
Charleston, WV 25325 



O 

ERLC 



3 



AEL's mission is to link the knowledge from research with the wisdom from practice to improve teaching 
and learning. AEL serves as the Regional Educational Laboratory for Kentucky, Tennessee, Virginia, and 
West Virginia. For these same four states, it operates both a Regional Technology in Education Consortium 
and the Eisenhower Regional Consortium for Mathematics and Science Education. In addition, it serves as 
the Region IV Comprehensive Center and operates the ERIC Clearinghouse on Rural Education and Small 
Schools. 

Information about AEL projects, programs, and services is available by writing or calling AEL. 




Post Office Box 1348 
Charleston, West Virginia 25325-1348 
304-347-0400 
800-624-9120 
304-347-0487 (fax) 
aelinfo@ael.org 
http://www.ael.org 



© 2000 by AEL, Inc. 



This publication is based on work sponsored wholly or in part by the Office of Educational Research and 
Improvement, U.S. Department of Education, under contract number RJ96006001. Its contents do not 
necessarily reflect the views of OERI, the Department, or any other agency of the U.S. government. 



AEL is an Equal Opportunity/Affirmative Action Employer. 



TABLE OF CONTENTS 



EXECUTIVE SUMMARY iii 

INTRODUCTION 1 

Background 1 

Content Validity 2 

Training Materials 2 

Dates, Locations, and Formats of the Training Sessions 3 

Purpose of the Training Sessions 3 

Audience for the Sessions 3 

Purpose and Audience for Report 3 

EVALUATION METHOD 4 

Participants 4 

Instrumentation 4 

Procedures 4 

FINDINGS 5 

Training 1: November 3, 1999 5 

Training 2: November 4, 1999 7 

Training 3: November 17, 1999 10 

Training 4: November 18, 1999 12 



i 



O 

ERIC 



5 



TABLE OF CONTENTS (continued) 



CONCLUSIONS AND RECOMMENDATIONS 15 

REFERENCES 16 

APPENDIXES 

A: Training Agenda 
B: Evaluation Form for Sessions 1 and 2 
C: Evaluation Form for Sessions 3 and 4 
D: Completed Evaluation Standards Checklist 

LIST OF TABLES 

1 : Descriptive Statistics for Training 1 Evaluation Form 6 

2: Descriptive Statistics for Training 2 Evaluation Form 8 

3: Descriptive Statistics for Training 3 Evaluation Form 11 

4: Descriptive Statistics for Training 4 Evaluation Form 13 



EXECUTIVE SUMMARY 



One of the mandates of the Kentucky Education Reform Act (KERA) of 1 990 is that students 
develop a writing portfolio. Ultimately, schools in Kentucky are expected to elevate the average 
rating of students’ portfolios to the benchmark of proficient. To aid in this endeavor, the Kentucky 
Department of Education (KDE) and AEL, Inc. began a collaborative research and development 
project in 1996 to help schools discern areas for improvement in their writing programs. A 
collaborative research team composed of KDE and AEL staff planned a series of site visits to 
ascertain the indicators of quality writing programs. Through site visits to 29 schools, the team 
identified 36 indicators that differentiated writing scores and programs in continuously improving 
schools from those in continuously declining schools. Based on these indicators, the team designed 
a needs assessment process instrument called the School Study of Writing Instruction (SSWI). This 
process, in the form of a handbook, enables a school to compare different role groups’ perspectives 
about the writing portfolio program. The SSWI was pilot tested in the fall of 1998 and field tested 
in the spring of 1 999. Experiences from these tests demonstrated that, while it was possible to 
complete the self-study without outside help, facilitator assistance was advantageous for schools 
carrying out the study. The collaborative team decided to offer facilitator training in the study 
process for educators who would be in a position to work with schools wishing to undertake the 
study. The purpose of the training was three-fold: to introduce the research findings and study 
process to persons in positions of instructional authority or influence in the state, to prepare partic- 
ipants to promote the use of the SSWI in schools, and to instruct participants in the tasks required 
to facilitate the study. The target audience included principals who were members of the Kentucky 
Leadership Academy, regional service center directors, state writing program staff, directors and staff 
of the state centers for the National Writing Program, district curriculum coordinators, and Highly 
Skilled Educators (HSEs, i.e., educational leaders designated to assist low-performing schools). 

Four sessions were held in 1999 to train individuals as potential facilitators of schools 
interested in conducting a SSWI. Each 1-day session constituted a complete training and was held in 
a location chosen from different geographic areas to accommodate educators from around the state: 
November 3, Convention Center Inn, Greenville; November 4, Northern Kentucky University, 
Covington; November 1 7, Jenny Wiley State Park, Prestonsburg; andNovember 1 8, Days Inn, London. 
Upon arrival, each participant received a copy ofthe self-study handbook — a process manual in a three- 
ring binder. The study includes three main steps: (1) interviews, (2) report, and (3) ratings and 
priorities. The interview questions, report topics, and rating categories correspond to the 36 quality 
indicators of improving students’ writing portfolio scores. They deal with issues of administrative 
support, professional development, school climate, writing program coordination, writing portfolios 
and writing in general, instructional strategies, family and community involvement, and assessment of 
the writing program’s value for students. The interviews are the data upon which all subsequent 
analyses are based. Each handbook also includes an introductory overview, a section for school 
planning, and facilitator notes. All handbook materials are provided both in print and on disk. 

The purpose of this report is to summarize evaluative data collected from participants at the 
training sessions. The primary audience includes staff of the U. S. Office of Educational Research 
and Improvement, AEL, and KDE. Other audiences include staff of the Kentucky Leadership 
Academy, regional service centers, national and state writing programs, HSEs, and other educators. 




in 



Altogether, 198 individuals attended one of the training sessions: 41 at session 1, 40 at 
session 2, 59 at session 3, and 58 at session 4. An evaluation form was developed that consisted of 
18 Likert-type items on a 5-point scale of satisfaction (1 = Not at all/Very little to 5 = Very much) 
with various aspects of the training related to content, presentation, and utility, as well as 2 open- 
ended response items soliciting insight into the areas of strength and weakness in the training. This 
form was administered at the first and second training sessions. The form was then revised to 
capture additional data about the handbook contents. The revised form consisted of 14 Likert-type 
items on the same 5-point scale of satisfaction and 3 open-ended response items. This version was 
administered at the third and fourth training sessions. The project coordinator attended each of the 
four training sessions to distribute and collect the forms at the conclusion of each. Data were entered 
into SPSS for Windows for analysis and qualitative data were categorized by theme. 

Findings. Response rates varied among the sessions: 83% of the participants completed the 
evaluation form for session 1, 82% for session 2, 58% for session 3, and 57% for session 4. 
Cronbach Alpha coefficients for each of the four administrations of the evaluation form were all at 
or above .90, indicating a high level of internal consistency among the items. Participants seemed 
very satisfied with various aspects of the training, as depicted by item ratings. For session 1, 15 of 
the 18 items had average ratings above 4.0; for session 2, 16 items had average ratings at or above 
4.0. For sessions 3 and 4, all 14 items had average ratings above 4.0. Participants viewed the 
handbook itself as the most beneficial part of the training, noting that it was well organized and self- 
explanatory. Their most frequent suggestion for improvement was to devote more time to the report 
writing and rating steps and less time on the interviews. Participants in sessions 3 and 4 viewed the 
SSWI material as helpful, beneficial, and useful. 

Conclusions and recommendations. Given participants’ ratings and comments, we 
conclude that the SSWI is viewed as a well-designed, research-based, and self-explanatory tool that 
schools could use in their improvement efforts targeting writing instruction. Respondents seem 
particularly pleased with the potential benefits of the process and feel the handbook has great utility 
for consolidation planning and needs assessment. We recommend that AEL continue investigating 
ways to market this product as a mechanism to help schools in their ongoing improvement efforts. 

The SSWI training sessions were well received and successfully accomplished their three-fold 
purpose of introducing the research findings and study process, preparing participants to promote 
use of the SSWI in schools, and instructing participants in the required tasks. Respondents seem well 
satisfied with the format and content of the training, in particular noting the usefulness of the 
material and their increased understanding of the SSWI. However, we recommend that staff inves- 
tigate and remediate for future trainings the circumstances that led to the low evaluation response 
rate for sessions 3 and 4, even though staff were on hand to collect the forms. Results for these two 
sessions may be skewed due to the high nonresponse rate. 

In general, participants seem to best understand step 1 of the SSWI process, which focuses 
on the interviews. Less well understood are step 2 (report writing) and step 3 (rating and setting 
priorities). This perception is reflected in numerous participant comments that too much time was 
devoted to the initial step and not enough time was provided for covering the later topics. Therefore, 
if future training sessions are offered, more time and focus should be given to these two critical steps. 

Finally, many participants agree that it would be more useful to receive the handbook before 
the training, which would allow them to review and become at least somewhat familiar with the 
materials in advance of the training. We recommend that the handbook be disseminated prior to the 
training and that participants receive copies of the overheads used during the training. 
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INTRODUCTION 



Background 

One of the mandates of the Kentucky Education Reform Act (KERA) of 1 990 is that students 
develop a writing portfolio (Kentucky Department of Education, 1990, 1998). Ultimately, schools 
in Kentucky are expected to elevate the average rating of students’ portfolios to the benchmark of 
proficient. To aid in this endeavor, the Kentucky Department of Education (KDE) and AEL, Inc. 
(formerly Appalachia Educational Laboratory, Inc.) began a collaborative research and development 
project in 1996 (AEL, 1997). A research team composed of KDE and AEL staff was formed. The 
goal of the project was to help schools discern areas for improvement in their writing programs. 

In 1996, the collaborative research team planned a series of site visits to ascertain the 
indicators of quality writing programs. Through site visits to 29 schools, the research team identified 
36 indicators that differentiated writing scores and programs in continuously improving schools from 
those in continuously declining schools. The indicators of successful writing programs address such 
issues as professional development, instructional strategies, administrative support, and family and 
community involvement. 

Based on the indicators that they had identified during the site visits, the collaborative 
research team designed a needs assessment process instrument called the School Study of Writing 
Instruction (AEL & KDE, 1999). This process, in the form of a handbook, enables a school to 
compare different role groups’ perspectives about the writing portfolio program. The comparison 
is made available to the entire faculty for school-wide awareness and discussion. Eventually, the 
results are used to set priorities and plan actions for improvement of a school’s writing program. 

A pilot test of the School Study of Writing Instruction was conducted in the fall of 1998 
(Parrish et al., 1999a) and a field test completed in the spring of 1999 (Parrish et al., 1999b). 
Experiences from the pilot and field tests of the School Study of Writing Instruction demonstrated 
that, while it was possible to complete the self-study without outside help, facilitator assistance was 
advantageous for schools carrying out the study. 

The collaborative team decided to offer facilitator training in the study process for educators 
who would be in a position to work with schools wishing to undertake the study. The training for 
potential facilitators was designed by members of the team who were former KDE writing 
consultants. They and other members of the research team from AEL met in July 1 999 to debrief 
about lessons learned from the pilot and field tests of the School Study of Writing Instruction. 
During this 2 Vz day meeting, the training designers began to plan the agenda for the training. They 
completed it during subsequent contacts over the next few weeks. At the same time, a team member 
developed facilitator notes based on field-test experiences. These notes were added to the process 
manual. 
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Content Validity 

Content validity of the School Study instrument was determined in two stages, each 
consisting of several steps. The first stage involved identifying the variables for examining writing 
programs (Coe et al., 1999a, 1999b). First, indicators were derived from the parameters of 
Kentucky’s writing program (see KDE, 1994). Second, the collaborative research team developed 
draft interview protocols around the initial indicators, tested them in exploratory visits to 7 Kentucky 
schools, and refined them. Third, the research team identified 43 schools with consistently 
improving writing scores and 18 with consistently declining scores. Based on site visits to 22 of the 
improving schools and 7 of the declining schools, in which more than 100 teachers, 200 randomly 
selected students, and 50 administrators were interviewed, team members established 36 indicators 
of successful writing programs (Coe et al., 1999a, 1999b). A comparison of the difference in mean 
scores between the 22 schools with improving scores and 7 schools with declining scores verified 
the indicators’ capability to differentiate writing programs. These steps encapsulate the process by 
which the indicators were validated. 

The second stage of validation involved creating the School Study of Writing Instruction as 
a process/instrument for school use. First, it was designed around the indicators identified in the 
research stage. Second, it was designed to foster the involvement of several school stakeholder 
groups in studying a school’s writing program, particularly with the inclusion of students as 
informants of writing instruction. Finally, to enhance utility, the School Study of Writing Instruction 
was revised through several iterations based on meetings of the collaborative research team, on the 
pilot test, and on the field test. 



Training Materials 

Upon arrival at the training site each participant received a copy of the self-study handbook 
that had been revised in light of the field test and that included the facilitator notes. The School 
Study of Writing Instruction (SSWI) handbook is a process manual in a three-ring binder. The study 
includes three main steps: (1) interviews, (2) report, and (3) ratings and priorities. The interview 
questions, report topics, and rating categories correspond to the 36 quality indicators of improving 
students’ writing portfolio scores. They deal with issues of administrative support, professional 
development, school climate, writing program coordination, focus on writing portfolios and writing 
in general, instructional strategies, family and community involvement, and assessment of the 
writing program’s value for students. The interviews are the data upon which all subsequent 
analyses are based. 

The handbook includes an introductory overview, a section for each of the three main steps 
listed above, and a supplemental section for school planning. Facilitator notes were added to the 
handbook after the field test was completed. All handbook materials are provided both in print and 
on computer disk. Handbook pages were copied and distributed so that participants could take a 
clean copy of the handbook with them at the conclusion of the training. 
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Dates, Locations, and Formats of the Training Sessions 

Four independent sessions were held in 1999 to train individuals as potential facilitators for 
schools interested in conducting a School Study of Writing Instruction. Each 1-day session 
constituted a complete training. An AEL staff member, who was part of the collaborative research 
team, coordinated planning the four sessions and supported the efforts of the trainers. Altogether, 
there were five trainers from the research team (some of whom had also designed the training) who 
divided the training responsibilities among themselves, two per session. 

Each of the four training sessions was held in a location chosen from different geographic 
areas to accommodate educators from around the state: November 3, Convention Center Inn, 
Greenville; November 4, Northern Kentucky University, Covington; November 17, Jenny Wiley 
State Park, Prestonsburg; and November 18, Days Inn, London. The training offered an overview 
of the study process and practice in its elements (see the agenda in Appendix A). Attendees were 
expected to assume their own costs for transportation and meals other than lunch, but received the 
training and materials at no charge. 



Purpose of the Training Sessions 

The purpose of the four sessions was three-fold: (1) to introduce the research findings and 
study process to persons in positions of instructional authority or influence in the state, (2) to prepare 
participants to promote the use of the SSW1 in schools, and (3) to instruct participants in the tasks 
required to facilitate the study. 



Audience for the Sessions 

Persons invited to the four sessions included principals who were members of the Kentucky 
Leadership Academy (which provides training similar to that of Highly Skilled Educators to build 
capacity for school improvement), regional service center directors, state writing program staff, 
directors and staff of the state centers for the National Writing Program, district curriculum 
coordinators, and Highly Skilled Educators (i.e., educational leaders designated to assist low- 
performing schools). 



Purpose and Audience for Report 

The purpose of this report is to summarize evaluative data collected from participants at the 
training sessions. The primary audience includes staff of the U. S. Office of Educational Research 
and Improvement, AEL, and KDE. Other audiences include staff of the Kentucky Leadership 
Academy, regional service centers, national and state writing programs, Highly Skilled Educators 
(HSEs), and other educators. 
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EVALUATION METHOD 



Participants 

Altogether, 198 individuals attended one of the training sessions: 41 at session 1 , 40 at session 
2, 59 at session 3, and 58 at session 4. The majority of the participants (1 1 3 , 57%) were school or district 
administrators (principals, assessment and curriculum coordinators, resource teachers, etc.). Participants 
from other role groups included HSEs (39, 20%), state administrators and consultants (1 3, 6%), university 
faculty (5, 2%), and others (classroom teachers or those without an identified role) (28, 14%). 

The collaborative research team invited people to the training in one of several ways. The 
principal means of invitation was a September 24 letter by then Commissioner William Cody, which 
was mailed to all HSEs, district personnel with responsibility for writing program oversight, and 
state and regional writing program leaders. The letter’s contents described the purpose of the 
training and included a registration form. AEL also sent invitations to state directors of the National 
Writing Program. And, the AEL project director contacted the leadership of the Kentucky Leader- 
ship Academy to alert that organization to the fact that principals were receiving the invitations. 



Instrumentation 

An evaluation form was developed by an evaluator member of the collaborative research 
team. It consisted of 1 8 Likert-type items on a 5-point scale of satisfaction (1 = Not at all/V ery little 
to 5 = Very much) with various aspects of the training related to content, presentation, and utility, 
as well as 2 open-ended response items soliciting insight into the areas of strength and weakness in 
the training (see Appendix B). It was submitted for comment to two other members of the research 
team prior to finalization. This form was administered at the first and second training sessions. 

After the first two trainings, a member of the research team, realizing that the evaluation 
questions addressed the training process only, created another version of the form. This version 
added an open-ended item about the handbook contents, i.e. the study process (“My opinion of the 
School Study of Writing Instruction materials is . . .”). In order to keep the evaluation form to one 
page, several other changes were also made: deleting question 9 and shortening former question 1 0 
in Section A, and deleting questions 3, 4, and 5 in Section B. The revised form consisted of 14 
Likert-type items on the same 5-point scale of satisfaction and 3 open-ended response items (see 
Appendix C). This second version was administered at the third and fourth training sessions. 



Procedures 

The coordinator attended each of the four training sessions to distribute and collect the forms 
at the conclusion of each. Data were entered into SPSS for Windows for analysis and qualitative 
data were categorized by theme. 
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FINDINGS 



Training 1: November 3, 1999 

Thirty-four out of 41 participants completed an evaluation form, representing a return rate 
of 83%. The Cronbach alpha coefficient for this set of scores for items in Sections A and B was .90, 
indicating a high level of internal consistency among the items. 

In Section A, participants responded to items that measured their satisfaction with various 
aspects of the training. The aspects of the training that they rated highest were provided materials 
that will be useful (mean of 4.85 on a 1-5 scale, standard deviation [SD] 0.36) and increased my 
understanding of the School Study of Writing Instruction (mean 4.79, SD 0.41). The lowest-rated 
items were was paced well (mean 3.88, SD 0.91) and was conducted in an engaging manner (mean 
4.32, SD 0.73). They rated the overall quality of the training as high (mean 4.53, SD 0.61). See 
Table 1 for more detail. 

In Section B, respondents answered items that assessed their understanding of the steps in 
the School Study of Writing Instruction. Participants rated highest two individual components of 
step 1: how interviewees are selected (mean of 4.68 on a 1-5 scale, SD 0.47) and how to prepare 
interviewers (mean 4.62, SD 0.55). Lowest rated were step 2, report writing (mean 3.88, SD 0.77), 
and step 3, rating and setting priorities (mean 3.84, SD 0.90). See Table 1 for more detail. 

In Section C, respondents answered two open-ended response items. In response to The most 
beneficial/interesting part of this training was . . ., 25 of the 34 respondents answered (9 with a 
multiple response), for a total of 36 responses. Seven respondents (19%) mentioned the materials 
or handbook, one saying that it was “well designed and self-explanatory.” Six (1 7%) made reference 
to the general format of the training, saying that they appreciated the overview or the process of 
going through the handbook. Five (14%) appreciated the research base behind the School Study of 
Writing Instruction, i.e., “the valid research based practice,” and “learning about the research and 
the 36 indicators.” Four (11%) named the SSWT s potential utility as an improvement tool for 
schools. Three (8%) liked the group activities or participant involvement. Three other respondents 
(8%) appreciated the examples given by presenters, i.e., “information/stories from your personal 
experiences with this process.” Two respondents (6%) complimented the presenters, specifically 
their flexibility to adjust to scheduling constraints or their skills to put the audience at ease, i.e., 
“They made you feel free to ask questions and express concerns.” 

The following themes merited one mention each (3% each): concise presentation; inclusion 
of student voice in the study process; discovery that the SSWI is process- rather than content-focused; 
the nonjudgmental, self-study aspect of the SSWI; and greater awareness of how to measure a 
school’s writing process. Also, one respondent mentioned a specific SSWI step — rating and setting 
priorities. 
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Table 1: Descriptive Statistics for Training 1 Evaluation Form 



Item 


N 


Mean* 


SD 


Section A 


1 . Was of good quality overall 


34 1 


4.53 


0.61 


2. Had clear outcomes 


34 


4.62 


0.60 


3. Was paced well 


34 


3.88 


0.91 


4. Had activities that reinforced the content 


34 


4.44 


0.66 


5. Was conducted in an engaging manner 


34 


4.32 


0.73 


6. Was conducted by knowledgeable and well-prepared presenters 


34 


4.71 


0.52 


7. Increased my understanding of the School Study of Writing Instruction 


34 


4.79 


0.41 


8. Increased my understanding of my role as a facilitator vs. other . . . roles 


34 


4.65 


0.54 


9. Enabled me to be capable of serving as a facilitator 


34 


4.44 


0.75 


10. Provided materials that will be useful to me in my future work as a facilitator 


34 


4.85 


0.36 


1 1. Stimulated me to want to use the skills and/or materials presented 


34 


4.56 


0.70 


Section B 


1 . How to introduce the study to a faculty and Steering Committee 


34 


4.38 


0.65 


2. How the interviews step is done overall (step 1) 


34 


4.59 


0.56 


3. How interviewees are selected 


34 


4.68 


0.47 


4. How interviewers are selected 


34 


4.56 


0.56 


5. How to prepare interviewers 


34 


4.62 


0.55 


6. How the report-writing step is done (step 2) 


34 


3.88 


0.77 


7. How the rating and setting priorities session is set up and conducted (step 3) 


31 


3.84 


0.90 



*On a scale of 1 (Not at all/Very little) to 5 (Very much). 



In response to This training could have been improved by . . 27 of the 34 respondents 

answered (9 with a multiple response), for a total of 37 responses. Twelve (32%) made a 
recommendation relative to the comparative time allotments per segment of the training or to the 
pacing. Eleven of these 12 stated that they would have preferred spending more time on report- 
writing and rating than the morning topics concerning interviewing: “maybe a little less time on the 
first steps (selecting interviews, etc.) and more on the end result”; “more time spent on the report and 
writing and less on the interviewing”; and “more focus on rating session and report writing.” Two 
of the 12 suggested starting the training with the rating step of the SSWI and working backwards and 
two others suggested a faster pace. 
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Eight respondents (22%) made a suggestion concerning the materials provided. Five of the 
eight specifically recommended that the various forms needed to be updated and revised so that they 
were aligned with each other in wording and numbering. One other respondent would have liked 
visuals in the handbook; also, this respondent thought that the SSWI did not consider writing beyond 
portfolio preparation. Another of these eight referred to pages 149-152 and suggested that the 
materials and comments should better address high schools. 

Four respondents (11%) suggested that more breaks be incorporated into the schedule, i.e., 
“More frequent breaks — sometimes the law of two feet is not exercised out of respect of presenters.” 
Two respondents (5%) would have liked more time for the training, both suggesting VA days; 
another two would have preferred receiving the materials prior to the training in order to review 
them. 



Each of the following themes received one mention (3% each): less time allotted for the 
training, frequent mini-reviews as the training proceeds, more direct participation, reconfiguring the 
groups after lunch, provision of copies of the overheads, provision of an agenda, and more 
comfortable facilities. Finally, two (5%) responded that they had no suggestions for improvement. 



Training 2: November 4, 1999 

Thirty-three out of 40 participants completed an evaluation form, representing a return rate 
of 82%. The Cronbach alpha coefficient for this set of scores for items in Sections A and B was .93, 
indicating a high level of internal consistency among the items. 

In Section A, participants responded to items that measured their satisfaction with various 
aspects of the training. The aspects of the training that they rated highest were increased my under- 
standing of the School Study of Writing Instruction (mean 4.67, SD 0.65) and provided materials 
that will be useful (mean 4.62, SD 0.61). The lowest-rated items were w paced well (mean 3.64, 
SD 1 .39) and was conducted in an engaging manner (mean 3.85, SD 1.18). They rated the overall 
quality of the training as moderately high (mean 4.18, SD 0.88). See Table 2 for more detail. 

In Section B, respondents answered items that assessed their understanding of the steps in 
the School Study of Writing Instruction. All items in Section B were rated moderately high or high, 
at or abo ve 4. 00 (SDs ranging from 0.61 to 1.15). Respondents had more understanding of the inter- 
views step than they did of the report writing or rating/priorities steps. See Table 2 for more detail. 

In Section C, respondents answered two open-ended response items. In response to The most 
beneficial/interesting part of this training was . . ., 23 of the 33 respondents answered (6 with a 
multiple response), for a total of 30 responses. Ten respondents (33%) named the handbook itself 
and/or other materials, i.e., the “study/binder itself — well thought out, organized, intentional — great 
resource for schools”; “the schedules and checklists made it less scary”; and “the hard copy examples 
given.” Six respondents (20%) made reference to the general format of the training, specifically the 
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Table 2: Descriptive Statistics for Training 2 Evaluation Form 



Item 


N 


Mean* 


SD 


Section A 


1 . Was of good quality overall 


33 


4.18 


0.88 


2. Had clear outcomes 


33 


4.55 


0.56 


3. Was paced well 


33 


3.64 


1.39 


4. Had activities that reinforced the content 


33 


4.06 


1.00 


5. Was conducted in an engaging manner 


33 


3.85 


1.18 


6. Was conducted by knowledgeable and well-prepared presenters 


33 


4.61 


0.56 


7. Increased my understanding of the School Study of Writing Instruction 


33 


4.67 


0.65 


8. Increased my understanding of my role as a facilitator vs. other . . . roles 


33 


4.39 


0.75 


9. Enabled me to be capable of serving as a facilitator 


33 


4.39 


0.79 


10. Provided materials that will be useful to me in my future work as a facilitator 


32 


4.62 


0.61 


1 1 . Stimulated me to want to use the skills and/or materials presented 


33 


4.09 


1.04 


Section B 


1. How to introduce the study to a faculty and Steering Committee 


33 


4.33 


0.74 


2. How the interviews step is done overall (step 1) 


33 


4.55 


0.62 


3. How interviewees are selected 


33 


4.55 


0.71 


4. How interviewers are selected 


33 


4.55 


0.71 


5. How to prepare interviewers 


33 


4.39 


0.61 


6. How the report-writing step is done (step 2) 


32 


4.06 


0.91 


7. How the rating and setting priorities session is set up and conducted (step 3) 


33 


4.00 


1.15 



*On a scale of 1 (Not at all/Very little) to 5 (Very much). 



process of chronologically examining the study process steps as outlined in the handbook, 
i.e., “hearing how the parts fit together in the process, understanding the how-to” and “the entire 
walk-through of the process.” Three respondents (10%) each named a specific step of the SSWI: 
interview questions, compiling results, and rating and setting priorities. Three (10%) appreciated 
the research base behind the SSWI, i.e., “thorough, clear materials based on solid research.” Three 
other respondents (10%) appreciated the examples given from the presenters’ personal experiences 
of conducting the study, i.e., “your enthusiasm and sharing of first-hand experience” and “hearing 
about experiences of those actually involved in the process — their tips were especially helpful.” 
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Two (7%) referred to the SSWFs potential utility as an improvement tool for schools. Two 
others (7%) liked the group activities or participant involvement. Finally, one respondent (3%) 
complimented the presenters’ skill, i.e., “[the] rapport of presenters with participants.” 

In response to This training could have been improved by 25 of the 33 respondents 
answered (7 with a multiple response), for a total of 36 responses. Eleven respondents (31%) made 
a recommendation relative to the comparative time allotments per segment of the training or to the 
pacing. Six of these 1 1 stated that they would have preferred spending more time on later topics 
than on earlier ones: “less time on first steering committee meeting and more time on report writing 
and setting priorities”; “skipping overviews, and getting to the meat of the process”; and “I’m still 
confused about the rating process.” Yet another respondent wanted more time spent on an earlier 
step, “more info [s/c] regarding selection of steering committee — this is crucial to the success of the 
study.” Four of the 1 1 suggested a faster pace overall, i.e., “moving at a faster pace” and “pick up 
pace — some areas you belabored too much.” 

Similarly, four respondents (11%) suggested that the training be conducted in less time, 
“collapsing the time — get me the info [s/c] and I can review it” and “cutting the time by at least 
half.” However, two others (6%) would have preferred more time, i.e. “just wish I had been able 
to spend another day with you learning!” 

One respondent (3%) made a suggestion concerning the materials provided, suggesting that 
the spelling and grammatical errors in the handbook be corrected. Two other respondents (6%) had 
a concern about the SSWI process itself, not the handbook or materials per se. One of these 
individuals believed that the study was not student-centered; the other thought that it was too 
complex — “Somehow the entire process needs to be simplified! Most schools would find this too 
cumbersome.” 

Two respondents (6%) recommended more participant involvement, i.e., “More direct 
participation — always hear, ‘If we had the time we would do . . .’ Perhaps we take that time”; and 
“The participants needed to do a ‘mock’ school study in order to internalize the complete process.” 
Also commenting about format, one respondent (3%) answered, “More practical — what will be 
faced — how to deal — how to cause quality work to happen.” One respondent (3%) recommended 
using larger overhead transparencies or providing copies of them because he or she could not see 
them adequately; one suggested playing a video for the audience; and one would have preferred 
receiving the materials prior to the training in order to review them. 

Also, there were two miscellaneous responses (3% each): one respondent would have liked 
more information about how to market the SSWI to a district and elicit its commitment, and another 
asked whether facilitators would be paid. 

Finally, eight respondents (22%) provided complimentary remarks or indicated that they had 
no suggestions. Sample comments include “Well constructed study and excellent materials” and “The 
presenters were motivating, and I look forward to facilitating a School Study of Writing Instruction." 
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Training 3: November 17, 1999 

Thirty-four out of 59 participants completed an evaluation form, representing a return rate 
of 58%. The Cronbach alpha coefficient for this set of scores for items in Sections A and B was .9 1 , 
indicating a high level of internal consistency among the items. 

In Section A, participants responded to items that measured their satisfaction with various 
aspects of the training. The aspects of the training that they rated highest were was conducted by 
knowledgeable and well-prepared presenters (mean 4.71, SD 0.52), increased my understanding of 
the School Study of Writing Instruction (mean 4.71, SD 0.52), and provided materials that will be 
useful (mean 4.71 , SD 0.58). The lowest-rated items were was paced well (mean 4.1 8, SD 0.80) and 
was conducted in an engaging manner (mean 4. 1 8, SD 0.83). They rated the overall quality of the 
training as moderately high to high (mean 4.47, SD 0.66). See Table 3 for more detail. 

In Section B, respondents answered items that assessed their understanding of the steps in 
the School Study of Writing Instruction. All items in Section B were rated moderately high to high, 
at or above 4.13 (SDs ranging from 0.62 to 0.78). Respondents understood the interviews step the 
most and the rating and priorities step the least. See Table 3 for more detail. 

In Section C, respondents answered three open-ended response items. In response to My 
opinion of the School Study of Writing Instruction materials is .... 30 of the 34 respondents 
answered (6 with a multiple response), for a total of 36 responses. Fifteen respondents (42%) 
referred to the SSWI as a helpful, beneficial, useful, or very useful tool for school improvement, i.e., 
“immediately useful”; “a very useful and timely tool”; “has much potential to improve school 
writing programs”; and “a quality program if a school has identified writing as a priority need.” One 
respondent (3%) felt that the SSWI was “Crucial to schools in our district. I’m not sure if we can pull 
this off because of PD [.sic] time this year, at least in some schools.” Four respondents (11%) 
described the SSWI as outstanding, valuable, or “good stuff!”; of these, one commented further: 
“Outstanding process! Should provide critical data for staff use. Follows best practice and adult 
learning principles.” Two respondents (6%) were less hyperbolic in their praise of the SSWI, saying 
that it was a good process. 

Six respondents (17%) actually described the usability of the SSWI materials themselves. 
Four of these used terms such as user-friendly, ready-to-use, or self-explanatory and easily followed. 
One said that it was well-organized. One said simply, “OK.” One isolated response (3%) indirectly 
referred to the usability of the materials, “The crucial part will be the sales at the beginning because 
of the committment [s/c] of time. It should not be rushed.” 

In addition, five other respondents (14%) commented on the SSWI’s utility, specifically for 
school consolidated planning and needs assessment, i.e., “useful, especially for consolidated 
planning” and “an effective self-evaluation tool.” One respondent (3%) suggested that it would be 
useful for grant-writing purposes. Finally, a miscellaneous response (3%) was “something that is 
growing — very informative,” but it was unclear whether the respondent meant the training in general 
or the materials specifically. 
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Table 3: Descriptive Statistics for Training 3 Evaluation Form 



Item 


N 


Mean* 


SD 


Section A 


1. Was of good quality overall 


34 


4.47 


0.66 


2. Had clear outcomes 


34 


4.59 


0.61 


3. Was paced well 


34 


4.18 


0.80 


4. Had activities that reinforced the content 


33 


4.42 


0.61 


5. Was conducted in an engaging manner 


34 


4.18 


0.83 


6. Was conducted by knowledgeable and well-prepared presenters 


34 


4.71 


0.52 


7. Increased my understanding of the School Study of Writing Instruction 


34 


4.71 


0.52 


8. Increased my understanding of my role as a facilitator vs. other . . . roles 


34 


4.56 


0.61 


9. Provided materials that will be useful to me 


34 


4.71 


0.58 


10. Stimulated me to want to use the skills and/or materials presented 


34 


4.50 


0.71 


Section B 


1 . How to introduce the study to a faculty and Steering Committee 


34 


4.35 


0.65 


2. How the interviews step is done overall (step 1) 


34 


4.50 


0.62 


3. How the report- writing step is done (step 2) 


33 


4.30 


0.64 


4. How the rating and setting priorities session is set up and conducted (step 3) 


30 


4.13 


0.78 



*On a scale of 1 (Not at all/Very little) to 5 (Very much). 



In response to The most beneficial/interesting part of this training was . ... 25 of the 34 
respondents answered (5 with a multiple response), for a total of 30 responses. Seven respondents 
(23%) named specific steps of the SSWI or parts of the SSWI handbook: five the practice in 
compiling interviews or the interview protocols themselves, one the practice in rating, and one the 
rating guide. Five respondents (17%) made reference to the general format of the training, 
specifically the process of chronologically examining the study process steps as outlined in the 
handbook: i.e., “the ‘guided’ explanations of the process”; “step-by-step, no gray areas”; “taking us 
all the way through the process with the materials”; and “going through the notebook and having the 
time to read the material as we went.” Four respondents (13%) commented on the whole training 
in general or increased knowledge from it, i.e., “The entire program was enchanting! I love it!” and 
“increased awareness of the writing program.” Three respondents (10%) named the handbook or 
materials. Two respondents (7%) appreciated the activities parts of the format, i.e., “Actually doing 
the processes discussed helped me most. The simulations gave me a deeper understanding of how 
to conduct this study.” One respondent (3%) appreciated the expertise of the presenters, i.e., “having 
presenters who have experienced the study.” 
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Eight respondents (27%) seemed to reflect on the SSWI process rather than on the training. 
Of these, six respondents commented on the SSWI’s utility for consolidated planning or needs 
assessment, one specifying planning for professional development needs. Two liked the depth or 
applicability of the study process, i.e., “very comprehensive process” and “actual school application.” 

In response to This training could have been improved by 14 of the 34 respondents 
answered (1 with a multiple response), for a total of 16 responses. Three respondents (19%) made 
a recommendation relative to the comparative time allotments per segment of the training or to the 
pacing, one specifying a preference to have spent more time on report writing and rating. Two 
(12%) commented on the handbook, one suggesting “a better organizer for compiling the report” and 
the other saying that the teacher interview is too long. Four other respondents (25%) expressed a 
wish for one of the following aspects of the format or training materials: more hands-on activities, 
tape transcripts to follow as listening, copies of overheads, and copies of handouts. 

Three respondents (19%) commented on the logistics of the training. Two suggested a 
change in room temperature and one suggested scheduling more breaks. One respondent (6%) 
wanted guidance on how to market the SSWI. Two respondents (12%) indicated that they had no 
suggestions. Finally, one response (6%) was idiosyncratic, the respondent saying that he or she did 
not realize that the event was to be a training session. 



Training 4: November 18, 1999 

Thirty-three out of 58 participants completed an evaluation form, representing a return rate 
of 57%. The Cronbach alpha coefficient for this set of scores for items in Sections A and B was .92, 
indicating a high level of internal consistency among the items. 

In Section A, participants responded to items that measured their satisfaction with various 
aspects of the training. The aspects of the training that they rated highest were was conducted by 
knowledgeable and well-prepared presenters (mean 4.88, SD 0.33) and provided materials that will 
be useful (mean 4.85, SD 0.36). The lowest-rated items were was paced well (mean 4.61, SD 0.66) 
and had activities that reinforced the content (mean 4.62, SD 0.61). They rated the overall quality 
of the training as high (mean 4.75, SD 0.51). See Table 4 for more detail. 

In Section B, respondents answered items that assessed their understanding of the steps in 
the School Study of Writing Instruction. All items in Section B were rated moderately high to high, 
at or above 4.30 (SDs ranging from 0.57 to 0.73). Respondents understood the interviews step the 
most and the report-writing step the least. See Table 4 for more detail. 

In Section C, respondents answered three open-ended response items. In response to My 
opinion of the School Study of Writing Instruction materials is . . ., 30 of the 33 respondents 
answered (5 with a multiple response), for a total of 36 responses. Twelve respondents (33%) 
referred to the SSWI as a very helpful, very beneficial, useful, or very useful tool for school 
improvement. One of these 12 commented, “Very beneficial to schools — especially those that are 
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Table 4: Descriptive Statistics for Training 4 Evaluation Form 



Item 


N 


Mean* 


SD 


Section A 


1 . Was of good quality overall 


32 


4.75 


0.51 


2. Had clear outcomes 


33 


4.79 


0.42 


3. Was paced well 


33 


4.61 


0.66 


4. Had activities that reinforced the content 


32 


4.62 


0.61 


5. Was conducted in an engaging manner 


33 


4.70 


0.59 


6. Was conducted by knowledgeable and well-prepared presenters 


33 


4.88 


0.33 


7. Increased my understanding of the School Study of Writing Instruction 


33 


4.79 


0.42 


8. Increased my understanding of my role as a facilitator vs. other . . . roles 


33 


4.67 


0.54 


9. Provided materials that will be useful to me 


33 


4.85 


0.36 


10. Stimulated me to want to use the skills and/or materials presented 


32 


4.66 


0.48 


Section B 


1 . How to introduce the study to a faculty and Steering Committee 


33 


*4.39 


0.66 


2. How the interviews step is done overall (step 1) 


33 


4.48 


0.57 


3. How the report-writing step is done (step 2) 


33 


4.30 


0.73 


4. How the rating and setting priorities session is set up and conducted (step 3) 


33 


4.36 


0.70 



*On a scale of 1 (Not at all/Very little) to 5 (Very much). 



high-achieving and needing to make those gains.” A lone respondent felt that the SSWI was more 
than just useful: “Very useful — vital if school is serious about improving writing.” Similarly, 
another 10 respondents (28%) described the SSWI as excellent, great, high, very good, or 
distinguished. 

Nine respondents (25%) actually described the usability of the SSWI materials themselves. 
Four of the nine called them very well organized, very well done, or well developed. Two said that 
the materials were clear and concise. Two described them as “manageable, useable” or “easy to 
follow.” One respondent said that the materials were attractive. 

In addition, four other respondents (11%) commented on the SSWI’s utility, specifically for 
school consolidated planning and needs assessment, i.e., “I like this. I believe it can give direction 
to staffs. It can be made a part of the needs assessment to our CP [sic].” 
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In response to The most beneficial/interesting part of this training was . . 27 of the 33 

respondents answered (3 with a multiple response), for a total of 30 responses. Six respondents 
(20%) named specific steps of the SSWI or parts of the SSWI handbook: three the rating guide, two 
the practice in rating, and one reading the samples. Five respondents (17%) commented on the 
whole training in general or the opportunity it provided for reflection, i.e., “It was all great”; “just 
understanding the overall picture of how it works”; and “reflection on writing instruction.” Five 
others (1 7%) simply noted the handbook or materials. One respondent (3%) made reference to the 
general format of the training, specifically the process of systematically examining the study process 
steps as outlined in the handbook, i.e., “walk through of process.” Another (3%) appreciated the 
activities parts of the format, i.e., “hands-on work with the materials.” In addition, one respondent 
(3%) said learning the duties of the facilitator was the most beneficial/interesting part of the training 
and another (3%) complimented the presenters as outstanding. 

Nine respondents (30%) seemed to reflect on the SSWI process more than on the training 
itself. Of these, four had comments about the mechanism of analysis inherent in the study: 

• The most interesting part is forming the perspective from the interview comments. 

• The fact that in the interviews teachers/students/administrators must self-evaluate their 
writing perceptions, practices, and beliefs. 

• Seeing the whole thing come together in a rating. 

• The insights gained by faculties as a result of having done this process. 

Two respondents commented on the SSWI's utility for consolidated planning or needs assessment. 
Three appreciated the research background to the creation of the SSWI, i.e., “the fact that indicators 
have been identified.” 

In response to This training could have been improved by 18 of the 33 respondents 
answered (2 with a multiple response), for a total of 20 responses. Two respondents (10%) made 
a recommendation relative to the comparative time allotments per segment of the training or to the 
pacing, with one specifying a preference to have spent more time on report writing and one 
specifying time for analyzing sample data. And, although two respondents (10%) suggested that 
more time be allotted for the training, another (5%) suggested that the training be done in less time. 

Regarding other format aspects, two respondents (10%) recommended more hands-on 
activities. One (5%) suggested a smaller group format and another (5%) would have preferred 
videotaped, rather than audiotaped, segments. Two (10%) commented on the handbook and/or 
materials, one stating that they needed to be current and the other stating that they needed more 
piloting. 

Four respondents (20%) commented on the logistics of the training. Three recommended a 
change in the facilities or room arrangement and one suggested a change in room temperature. 
Finally, five respondents (25%) had no suggestions for improving the training. 




22 



15 



CONCLUSIONS AND RECOMMENDATIONS 



Given participants’ ratings and comments, we conclude that the School Study of Writing 
Instruction is viewed as a well-designed, research-based, and self-explanatory tool that schools could 
use in their improvement efforts targeting writing instruction. Respondents seem particularly 
pleased with the potential benefits of the process and feel the handbook has great utility for 
consolidation planning and needs assessment. We recommend that AEL continue investigating ways 
to market this product as a mechanism to help schools in their ongoing improvement efforts. 

We conclude that the four SSWI training sessions were well received and successfully accom- 
plished their three-fold purpose of introducing the research findings and study process, preparing 
participants to promote use of the SSWI in schools, and instructing participants in the tasks required 
to facilitate the study. Respondents seem well satisfied with the format and content of the training, 
in particular noting the usefulness of the material and their increased understanding of the SSWI. 
However, we recommend that staff investigate and remediate for future trainings the circumstances 
that led to the low evaluation response rate for sessions 3 and 4, even though staff were present to 
collect the forms. Results for these two sessions may be skewed due to the high nonresponse rate. 

In general, participants seem to best understand step 1 of the SSWI process, which focuses 
on the interviews. Less well understood are step 2 (report writing) and step 3 (rating and setting 
priorities). This perception is reflected in numerous participant comments that too much time was 
devoted to the initial step and not enough time was provided for covering the later topics. Therefore, 
if future training sessions are offered, more time and focus should be given to these two critical steps. 

Participants seem to have differing viewpoints on the timing or pacing of the training 
sessions. In each of the sessions except for the third, some respondents requested extending the 
training to more fully cover the content, others felt the session should be shortened and the pace 
increased, and still others were content with the training as provided. With such a diversity of 
opinions, it seems that the length of the training lies somewhere in the “happy medium” and satisfies 
the majority of participants. Therefore, we recommend not altering the one-day time frame for future 
trainings. 

There is a general consensus among participants that the materials are well organized and 
easy to use, though some feel the handbook needs to be finalized. Several participants suggested the 
handbook materials be updated and typographical errors corrected. Therefore, we recommend the 
SSWI handbook be carefully reviewed and revised before any further dissemination takes place. 

Finally, many participants are in agreement that it would be more useful to receive the 
handbook prior to the training. This would allow participants to review and become at least 
somewhat familiar with the materials in advance of the training. We recommend that the handbook 
be disseminated prior to training and that participants receive copies of overheads used throughout 
the training. 



O 

ERiC 



23 



16 



REFERENCES 



AEL. (1997). AEL activities in Kentucky, 1996-97. Unpublished manuscript. 

AEL & Kentucky Department of Education. (1999). School Study of Writing Instruction: Field test 
version. [Process manual]. Charleston, WV: AEL. 

Coe, P., Keyes, M., Lewis, S., Meehan, M., Orletsky, S., Rigney, S., Runge, C., & Whitaker, J. 
(1999a). Development and validation of successful writing program indicators based on 
research in continuously improving and continuously declining schools. Paper presented at the 
meeting of the American Educational Research Association, Montreal, Canada. 

Coe, P., Keyes, M., Meehan, M., Orletsky, S., Lewis, S., Rigney, S., Runge, C., & Whitaker, J. 
(1999b). Development and validation of successful writing program indicators based on 
research in continuously improving and continuously declining schools: Report of the Kentucky 
state writing project. Charleston, WV : AEL. 

Kentucky Department of Education. (1990). 1990 Kentucky school laws. Frankfort, KY: Author. 

Kentucky Department of Education. (1994). Kentucky writing portfolio: Writing portfolio 

development teacher ’s handbook. [2nd ed.]. Frankfort, KY : Author. 

Kentucky Department of Education. (1998). 1998 Kentucky school laws. Frankfort, KY: Author. 

Parrish, P., Keyes, M., Coe, P., Tolbert, S., Orletsky, S., Lewis, S., Runge, C., Meehan, M., & 
Whitaker, J. (1999a). Pilot test of the School Study ofWriting Instruction : A self-study needs 
assessment instrument. Charleston, WV: AEL. 



Parrish, P., Keyes, M., Orletsky, S., Coe, P., Runge, C., Meehan, M., Whitaker, J., Nickell, M., 
Roberts, J., Sallee, M., Ladd, P., Caudill, C., Foster, G., Hatton, S., Lewis, S., & Tolbert, S. 
(1 999b). Field test of the School Study ofWriting Instruction : A self-study needs assessment 
instrument. Charleston, WV: AEL. 





APPENDIXES 



Appendix A: 
Training Agenda 



School Study of Writing Instruction 
Facilitator’s Guide 



1 



NOTE: Be sure to provide 5 or so sign-in sheets for participants so that activity during student selection 
portion can be implemented. 



I. Introduction (30 minutes) 8:30 

Welcome and Housekeeping (5) 

• Guidelines for day, parking lot, breaks, etc. 

• STRESS that this is a FULL day's training, that leaving early will diminish the 
effectiveness of the objectives, and those who leave prior to its completion 
will not understand the final steps as well as the true value of the process. 

Purpose/Goals (5) 

• To prepare external facilitators with the information and skills necessary to 
assist school personnel with implementation of the School Study of Writing 
Instruction 

Objectives 

• Participants wiH~- 

1 . Examine collaborative research by the Kentucky Writing Program and 
the Appalachia Educational Laboratory 

2. Become familiar with School Study of Writing Instruction (SSWI) handbook 

** Participants should make notes as we work through materials 

3. Understand and acquire facilitation strategies to assist schools through 
the process and procedures 

OH 1 - Purpose and Objectives 
OH 2 - Core Components of Training 

**4 Interrelated training components: Knowledge, Demonstration, Practice, 

Feedback 

History of KWP (2) 

OH 3 and Handbook 

Information about AEL (3) 

OH 4 and Handbook 

History of Research and Collaboration (3) 

OH 5 -Timeline 

Facilitator’s Role (12) 

• Use “Nominal Group Process” to identify important roles of a skillful 
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Facilitator OH 6 

1 . Each participant writes ideas on individual post-its (1 idea per post-it) 
for trainer to share and post on chart. 

2. Trainer writes the following categories on chart. 

■ confidentiality 

■ school’s sense of ownership 

■ facilitation, rather than judgment 

■ difference between audit/evaluation and facilitation of self-evaluation 

II. Overview of SSWI Handbook (40 minutes) 9:00 am 

Predicting Activity (10) 



(Remind participants that all discussions will be quick (like quick writes) 1 st thoughts, due to 
time constraints) 



• Using OH, work together to predict answers. 

OH 7A - Questions from p. 1 SSWI handbook: 

Why undertake a school study of writing instruction? 

Who might benefit from this study? 

What does the study process involve? 

How long will it take to complete the process? 

□ OH7B 

Small groups will: 

1 . Predict answers to questions before opening manual. 

2. Read text to determine agreement (p. 1 of Overview). 

3. Revise answers. 

□ Whole group - brief discussion. 

• Preparing for the Task (7) 

□ Ask 3 individuals to read #1, #2 and A-E "Preparation for Undertaking 
the Study", page 2, with trainer emphasizing the following points for 
the entire group: 

4. “selling” for faculty buy-in 

5. makeup of the steering committee (not too many) 

6. following timeline (not spread out over too much time) 



**Refer to Facilitator's Notes in handbook: participants need to insert into their handbooks according to 
directions at top of these notes to read later as a review^ summary before working^ with school 



• Materials Needed (1) 

□ Reminder of research information from earlier activity (history of 
project, etc.) 
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□ Importance of each committee member having handbook. 

□ Emphasize our role today of familiarity with handbook 

• Steps in Process (1) 

OH 8 - 4 Steps (p. 3) 

□ Participants read - in-depth discussion will occur later in training 

• Time Required (8) 

OH 9 - bottom of p. 3 and refer to chart, p. 4 (Summary of time required) 

□ Participants review time and task summary independently. 

1 . Use post-its to write questions as information is reviewed (attach 
Post-its on edges of that page so can refer to them later). 

2. Initiate thinking about passage, no answers at this time 

• Duties of the Steering Committee (13) 

□ Participants read silently to discover the responsibilities of the steering 
committee and highlight important information from A - H (p. 7). 

1. A-F Planning Meeting and Data Gathering 

2. G-H Report Writing (most time-consuming) and Rating/Planning 
Meeting 

OH 10 - Steering Committee Responsibilities 
OH 1 1 - Summary on p.8 for Review. 



**Re-emphasize at this point that the information we have covered thus far (the overview 
information and first four pages) is the information to be presented to the school community 
during the selling/ownership meeting and/or with the steering committee at its first planning 
meeting. (Go to page 5) 



III. Interviews - The raw data on which everything else is based 
9:40 

Preparing for the Interviews (10) 

Using OH 12, discuss how to help steering committee prepare for interviews: 

□ Select teachers to be interviewed 

□ Select students to be interviewed 

□ Select central office administrator to be interviewed 

□ Identify and prepare people to conduct interviews with teachers, 
students, and administrators 

□ Prepare to conduct interviews 

□ Designate a person to receive/collect interviews 



• Importance of Random Selection (10) 
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□ Using OH 13 -Purpose of Random Selection, discuss paragraph 1 on 

p. 15 in handbook. 

1 . Learn whether teachers have understanding of how to implement 
writing instruction 

2. Discover how students are responding to writing instruction 

3. Information for “your” school 

4. Be accurate in your school’s “picture” (No need to select those 
who can present “best” picture) 

□ Read/discuss 2 nd paragraph 

1. Add scenarios/ “What ifs” using OH 13a 

2. Refer to examples from “Stories from Kentucky Schools” in back 
of handbook. 



** Emphasize student voice and randomness 



BREAK 



10:00 



2. NOTE: TIME will be a big factor in this section. 



• Selecting Teachers to be Interviewed (12) 10:10 

□ Small Group Activity With "What Ifs" 

Use OH 14 to describe the task (trainer cut "what ifs into strips ahead of 
time and give one to each group) 

1 . Choose a recorder, reporter, timer, and facilitator in each group 

2. Read your group’s "What if?" 

3. Read the “Teacher Selection Method” sections on pp. 15 through 
top of 1 7 and then to top of 1 8 

4. Group discusses the "what-if 1 in relation to your reading (quick 
talk) 

5. Determine whether or not the selection method is appropriate or 
inappropriate. 

6. Explain why or why not 

□ Whole Group Activity (POSSIBLY OMIT IF RUNNING BEHIND) 

Each group reports briefly for whole group. 

On overhead or chart, trainer highlights important points made by 
group reporting. 

• Selecting Students to be Interviewed (13) 



**Note: Trainer will do this in first steering committee meeting. See page 11-1 through 
11-3 in Facilitator's Notes (to be inserted before page 5 in handbook) and beginning on 
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the bottom of page 17-18, and 19-20 in handbook. 



• Stratified Random Sampling Activity: Simulation of student 
selection. 

□ Follow the procedure given on student interviews, pp. 17-20. 

1 . Use one table of participants to represent the steering 
committee. 

2. Use the 5 morning sign-in sheets to represent class lists of 
selected teachers. 

3. Place pre-selected numbers (see p. 19) in a container. 

4. Ask a “steering committee member” to draw a number from 
“Teacher #1 ’s” class list. 

5. Locate the name on that sheet. 

6. Another “steering committee member” records the name 

7. Return numbered slip of paper to the container. 

8. Repeat steps 2-7 and continue drawing until one student from 
each selected teacher (i.e., the morning sign-in sheets) is 
identified 

9. Return to first class list. Repeat the process to model HOW to 
continue random selection from class lists. 



** Role-play: Horror about some student names selected, etc. 



1. OH 15 - Emphasize: 

• Identify MORE than needed. 

• Repeat - This is MINIMUM number of students to select. 
To interview MORE is great but may complicate report 
writing. 

• Emphasize MUST be random. 

2. After simulation, review, read and highlight procedure in 
handbook, pages 11-12-13. 

• Reward a candy bar to each person whose name is selected. 

• Selection of Interviewers/Teams (15) 10:35 

□ Individually, highlight important points to remember from 
“Interviewer Selection Guide” top of page 1 1 in handbook. 



** Note: Emphasize selection of interview TEAMS. 

□ OH 16 Whole Group/Small Group Activity 

1 . Identify main points on page. 

2. Brainstorm in small groups (2 minutes) to create on a post-it a 
list of persons from own school community who might make good 
interviewers. Recall the suggestions on the page for selection. 
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3. Each group shares one not previously shared and explains why 
that person was chosen. 

4. Chart as whole group shares out to create one list 

5. Facilitator will reinforce or share others which worked well for 
field-test schools. (Have chart prepared from field-test pilot for 
other information and compare.) 

6. Collect and display post-its for later review. 

□ Additional question: Who might not be successful? 

OH 17 - Address briefly these scenarios: 

• Teacher interviewing another teacher? 

• Parent interviewing own child? 

• Teacher from previous year interviewing student, etc. 

After each ask, "Could this be a problem? How?" 

• Preparation of Interviewers (20) 10:50 

□ Read for key points in “Interviewer Preparation Guide” and “Training 
Interviewers,” pp.l 1-13 in handbook. 



NOTE: If time is short, cut out the following two activities, beginning again with 
"Teacher Interviews" and summarize. 



□ Create, with a partner, a graphic to share this information 
OH 18 - Directions for preparing the graphic 

1 . Read for key points 

2. Consider how you would best organize and summarize this 
information 

3. Create, with a partner, a draft of a graphic to share this information 
with those interviewers selected at your designated school 

4. Post the draft of your graphic on the wall to share 
**Materials needed: typing paper, markers, chart paper 

□ Do Carousel Walk at lunch. Participants write name on post-it for a 
copy of any graphic to be used at their school(s). 



• Teacher interviews (20) 11:10 

□ Participants will assume the role of an interviewer preparing for a teacher 
interview. 

1 . Read “Teacher Interview Instructions,” on p. 22 in handbook, review 
the checklist and materials list on page 22 and scan the ‘Teacher 
Interview Schedule” on p. 21. 
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**Use OH 20 - Checklist, Materials List (p. 22) and OH 21 - Teacher 
Interview Schedule (page 21) to show participants handbook pages to 

consult. 

2. Jigsaw in small groups to review the 50 questions on the teacher 
interview form. 

3. Participants place Post-its with possible questions in appropriate spots. 

4. Share possible questions in whole group. 

**As trainers we must be prepared to answer and know possible trouble spots and areas 
for probing. 



a Trainers will role-play (model) an interview for 2 or 3 questions 
Ask a participant to serve as scribe on overhead and co-questioner. 

Lunch 11:30 — 12:30 (Maybe - take less time if running behind) 

• Preparing Student Interviewers (20) 12:30 

□ Participants will assume the role of an interviewer preparing for a student 
interview. (10) 

OH 22 

1 . Read “Student Interview Instructions,” on p.36 in handbook 

2. Review the checklist and materials list 

3. Scan the “Student Interview Schedule” on p.35. 

**Use OH 23A and OH 23B - Checklist, Materials List, Student 
Interview Schedule- to show participants the pages to consult. 

4. OH 23 C Point out differences in teacher and student interviews: 

■ Work with a partner 

■ Put students at ease, explain process, etc. 

■ Interview 2 or 3 students together 

■ If question is not understood, rephrase or give an example 

□ Share copy of script (or play tape) from AEL of excerpt from student 
interview which shows “probing.” 

• Review student interview questions on pages 37-44 (15) 12:50 

q Compare to teacher interview questions 

5. Direct participants to note differences/similarities of questions on 
post-its 

6. Lead brief whole group discussion pointing out questions which 
might require probing 



**Note: May require some modeling by trainers. 
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• Preparing for Principal and Central Office Interviews (10) 1 :05 

□ OH 24 Direct participants to highlight the following points on pages 
45 and 55 in their handbooks : 

1. Who will do these interviews? 

2. How will these likely be different? 

3. What problems might arise? 



** Emphasize confidentiality 



□ OH 25 Quickly scan : 

1. Principal Interview Instructions on p 45 of handbook. 

2. District Administrator Interview Instructions on p. 55 

3. After reading, have participants note the differences in opening 
paragraph of each 

4. Discuss these differences 

• Brief whole-group summary (5) 

□ Tips for interviewer {Refer to Facilitator Notes, 111-1 through 111-3, 
and have participants insert before page 9 in handbook) 

1. Some interviewees will tell you false perceptions. WTiattodo? 

2. Write what they say? 

3. Watch interviewer’s body language 

4. Discuss scheduling of interviews - leaving time between, avoiding 
lunch, etc. 

• Review of Duties and Meeting #1 (5) 1:20 

□ Return to OH of Steering Committee Duties) OH 10: Note: We've 
covered A -F. 

□ Use OH 26 of Sample Agenda for First Steering Committee 
Meeting (page 5) 

IV. Developing the Report (Planning Meeting #2) (45) 1 :25 

• Compiling information from the interviews (15) 

□ Distribute compiled interview response forms excerpts 

1. Note the way information has been entered (by a typist) for ease of 
report writing. 

2. Point out how comments are coded by role (4 th grade 
student, content area teacher, etc.) and for confidentiality. 



***Refer to bottom of page 61 and 62 to remind interviewers to identify teachers by 
grade level/ and whether teachers are content (not SS or PE) or LA teacher, and 
students by grade levels. 



1 . Emphasize that typist makes no judgment calls but types 




34 



9 



only what each interviewee said as recorded on the 
individual interview forms 

2. Emphasize “Other” section for adding extra information. 
**See Directions for Compiling Interview Responses, pp. 61 and 62 in handbook for 
further explanation/clarification. 

BREAK 1:40-1:50 



• Writing the Report (25) 1:50 

□ Using the table on pages 69-72 and the report form on pages 73-94, 
simulate selection of committee members to develop each of the 8 report 
sections. 

1 . Participants will assume the roles of committee members 

2. Participants will examine the section of sample completed reports 
for which they are responsible (both a thorough one — copy is in 
sleeve of each handbook — and a lean one (pilot school report from 
field study materials) — trainer puts one copy on each table made 
on colored paper for easy retrieval after this activity by trainer). 



*** Note: trainer may also want to make OH of one or two lean pages so all can see! 



1 . Facilitator will lead discussion of the advantages/disadvantages of 
the two samples in achieving the goals of the study. 

□ Review directions for writing the report (pages 67-68) 

1 . Facilitator uses OH 27 to talk through process 

2. Insert pages IV from Facilitator Notes before page 63 in handbook. 

□ Practice writing the report 

1 . Participants will write one section of the report. For discussion 
purposes, all participants will write the same section (9 A) 

2. Whole group shares one or two reports and discusses any possible 
difficulties which may arise. 



**Purpose of this task is to learn procedure for writing report and to think through 
possible questions. 



□ Review of Planning Meeting #2 2:15 

1. Use "Steering Committee Meeting to Prepare to Write Report," 
page 63, to lead discussion and answer questions. 

Refer also to pages 137-138: "Indicators that distinguish 
consistently improving from consistently declining schools" 
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III. Preparing for the Meeting to Plan the Faculty Meeting to Assess 
the Writing Program and to Determine Priorities for Action 
( 45 ) 2:25 



**Trainer emphasizes this as most challenging part of the process. 

• Organization is key. 

• Proper preparation is urgent. 

This meeting has the potential for creating a collaborative, team atmosphere 
among facult y OR for causing division/con fusion. 



• Understanding the meeting process (15) 

□ Give each table a fictitious school information sheet (Handout) to use for 
the following activity: 

1. Use OH 28— questions and directions to guide participants in the 
activity. 

2. Participants will read about the rating meeting process on pages 97-99 
to answer the following questions: 

■ Who is reviewing the report for your school? 

■ How are the groups organized? 

■ Are the group sizes as recommended in the guide? 

■ If not, why not? 

■ Are the indicator numbers assigned fairly? 

■ Were the participants given an opportunity to review the report 
before the meeting? 

3. Be prepared to share your information with the whole group and 
support your answers with information from the manual. 

4. Find in Facilitator's Notes page V; insert before page 95 in handbook. 

□ Go over Checklist for Rating/Planning meeting on page 95. 

1. Use OH 29 to review the preparations for the meeting 



** Emphasize (1) ''Organization is the key " AND (2) Properpreparation is essential 

• Steps in the Rating Process (20) 

□ Read and discuss rating directions on p. 104 of the handbook. 

□ Using a sample report, divide it into sections and assign one section to 
each group. Have participants will attempt to complete the rating guide 
for that school. Follow the directions just read and discussed. Emphasize 
any difficulties they may have. 

Use OH 30 to provide the following directions: 
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1 . Rate individually 

2. Get a group consensus 

3. Chart findings on graph 

4. Share with whole group 



IF TIME IS SHORT: Use only section 9A on page 120 and rate that indicator. 



• Procedure for preliminary setting of priorities (10) 

□ Directions for group walk-around 

1 . Each participant will be given a colored dot. 

2. In a walk-around, participants will use the colored dot to identify what 
they consider the most crucial areas for action 

3. Whole group will then reach consensus on those areas of greatest need 

4. After reaching consensus, recommendations will be made to CP 
committee for action or another meeting time can be set to determine 
how to meet these needs. 



IF TIME IS SHORT: 

1 . On page 1 34, look at graph of low profile or high profile school. 

2. Choose several possible priorities for choices, discussing why some would be better 
than others. 



• Review of Rating Planning Meeting (5) 

□ Use OH of Agenda for Rating/Planning Meeting (see Handbook pp. 101- 
103 and handouts for related information) to review this meeting. Note 
the points given at the beginning of this section. 



VI. 



Reflection/Evaluation 



3:10 



Appendix B : 

Evaluation Form for Sessions 1 and 2 
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Evaluation of the 

Facilitator Training in the School Study of Writing Instruction 



Thank you for your participation in this training. Your evaluation will help us continually improve our 
services to meet your professional needs. Please take a few moments to tell us what you thought of the 
training. Your responses will remain anonymous and confidential. 



Section A: Circle the number that best indicates the extent to which this training 



1 . 

2. 

3. 

4. 

5. 

6 . 

7. 

8 . 

9. 

10 . 
11 . 



Not at all 

Was of good quality overall. 1 2 3 

Had clear outcomes. 1 2 3 

Was paced well. 1 2 3 

Had activities that reinforced the content. 1 2 3 

Was conducted in an engaging manner. 1 2 3 

Was conducted by knowledgeable and 

well-prepared presenters. 1 2 3 

Increased my understanding of the School Study 

of Writing Instruction. 1 2 3 

Increased my understanding of my role as a 

facilitator versus other, evaluative roles. 1 2 3 

Enabled me to be capable of serving as a 

facilitator. 1 2 3 

Provided materials that will be useful to me in 

my future work as an facilitator. 1 2 3 

Stimulated me to want to use the skills and/or 

materials presented. 1 2 3 



4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 



Very much 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 

5 



Section B: Circle the number that best represents your current level of understanding about . . . 



Very little 

1 . How to introduce the study to both a faculty 

and a school’s Steering Committee. 12 3 4 

2. How the interviews step is done overall (step 1). 12 3 4 

3. How interviewees are selected. 12 3 4 

4. How interviewers are selected. 12 3 4 

5. How to prepare interviewers. 12 3 4 

6. How the report-writing step is done (step 2). 12 3 4 

7. How the rating and setting priorities session is set up 

and conducted (step 3). 12 3 4 

Section C: Please respond to the following prompts (use reverse side if needed). 

1 . The most beneficial/interesting part of this training was . . . 



Very much 

5 

5 

5 

5 

5 

5 



This training could have been improved by . . . 
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Thank you! This training was presented by the Kentucky Department of Education and AEL, Inc. 



Appendix C: 

Evaluation Form for Sessions 3 and 4 



Evaluation of the 

Facilitator Training in the School Study of Writing Instruction 

Thank you for your participation in this training. Your evaluation will help us continually improve our services to 
meet your professional needs. Please take a few moments to tell us what you thought of the training. Your 
responses will remain anonymous and confidential. 

Section A: Circle the number that best indicates the extent to which this training . . . 



Not at all Very much 



1. 


Was of good quality overall. 


1 2 3 


; 4 


5 


2. 


Had clear outcomes. 


1 2 3 


4 


5 


3. 


Was paced well. 


1 2 3 


4 


5 


4. 


Had activities that reinforced the content. 


1 2 3 


4 


5 


5. 


Was conducted in an engaging manner. 


1 2 3 


4 


5 


6. 


Was conducted by knowledgeable and 










well-prepared presenters. 


1 2 3 


4 


5 


7. 


Increased my understanding of the School 










Study of Writing Instruction. 


1 2 3 


4 


5 


8. 


Increased my understanding of my role as a 










facilitator versus other, evaluative roles. 


1 2 3 


4 


5 


9. 


Provided materials that will be useful to me. 


1 2 3 


4 


5 


10. 


Stimulated me to want to use the skills and/or 










materials presented. 


1 2 3 


4 


5 


Section B: Circle the number that best represents your current level of understanding about . 








Very little 


Very much 






1 2 


3 4 


5 


1. 


How to introduce the study to both a faculty 










and a school’s Steering Committee. 


12 


3 4 


5 


2. 


How the interviews step is done overall (step 1). 


1 2 


3 4 


5 


3. 


How the report-writing step is done (step 2). 


1 2 


3 4 


5 


4. 


How the rating and setting priorities session is set up 










and conducted (step 3). 


1 2 


3 4 


5 



Section C: Please respond to the following prompts (use reverse side if needed). 

1. My opinion of the School Study of Writing Instruction materials is: 



2. The most beneficial/interesting part of this training was . . . 



3. This training could have been improved by . . . 



Thank you! This training was presented by the Kentucky Department of Education and AEL, Inc . 






Appendix D: 

Completed Evaluation Standards Checklist 



Checklist for Applying the Standards 

To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


X 








U3 Information Scope and Selection 


X 








U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 


X 








U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 


X 








F3 Cost Effectiveness 


X 








PI Service Orientation 


X 








P2 Formal Agreements 


X 








P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 








P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








A10 Justified Conclusions 


X 








All Impartial Reporting 


X 








A 12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

£ evaluation report 

other: 



Name Kimberly S. Cowley 

fl/j/ 






;trear 



(signaturi 

Position or Title Research Associate < 



Agency 

Address 



AEL, Inc. 



Post Office Box 1348, Charleston, WV 25325 



Relation to Document 



Co-author 



(e.g., author of document, evaluation team leader, external auditor, internal auditor) 



July 2000 



ERJC 
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